Skip to content

Fix the 4.11 review findings across detection, guardrails and releases - #187

Merged
sidmohan0 merged 26 commits into
devfrom
fix/review-4.11-remediation
Nov 16, 2026
Merged

sidmohan0 merged 26 commits into
devfrom
fix/review-4.11-remediation

Conversation

@sidmohan0

@sidmohan0 sidmohan0 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Structured PII now uses the same backend across regex, smart, spaCy, GLiNER, CLI and legacy entrypoints. Lone surrogates preserve source offsets, NER uses real spans without dropping structured findings, TextService covers chunk boundaries, and dense results and fallback patterns avoid the reported quadratic behavior. Guardrails await async results and cover tool arguments, prompt/input shapes and complete buffered streams with request-scoped tokens.

The remaining review changes repair pipeline scan defaults, algorithm selection and optional HMAC, allowlist timeouts, entity validation, Pydantic/anonymizer/Spark behavior, CLI secret output and packaging. Existing plain-hash identities and empty legacy anonymizer filters stay compatible. Scheduled releases qualify without publication; manual runs default to dry runs, reject stale histories and occupied versions, and record validated refs before upload. The branch also brings the stable changes missing from dev into development.

Local validation includes the non-slow suite, installed-wheel smoke, frozen/native contracts, all pre-commit hooks and actionlint. The real spaCy/GLiNER native corpus passed 526 cases with 45 existing expected failures and zero skips; only two exact URL spans are newly permitted. Historical 4.8.1 observations and prior reviewed Core 1.4 fixtures are unchanged. Every Python finding has its resolution and evidence in docs/review-remediation.md.

Core 1.5.0 is the separate unpublished minor-release candidate; public APIs and wire contract 1 stay available. This parent PR intentionally retains the published Core 1.4 minor pin for standalone CI. The dependent release update raises it to ~=1.5.0 and must pass installed published-artifact qualification after Core publication. No publication is performed.

release: DataFog Python 4.9.0
Resolve backend through 'auto': the native engine when datafog-core
imports, the pure-Python engine otherwise with a single warning.
DATAFOG_BACKEND and backend="python" force the fallback; non-regex
engines stay on the Python ML path.

Serve prose PERSON and ADDRESS from the embedded model in datafog-core,
so names and addresses need no spaCy, GLiNER, download or interpreter.
Expose the new credential, account and UUID entity types through the
alias API and add datafog.native for the native schema.

Deprecation warnings now name 4.10.0 as the last supported version with
an exact pin, and VERSIONING.md plus docs/retirement.rst make the window
and the removal schedule auditable. Nothing is removed in this release.
1.2.0 shipped without the exchange-code fix, so 4.10.0 depends on the
release that has it. Contract file follows the engine version.
The corpus encodes Python-engine expectations, so it pins
backend=python by default and scores the native engine under
DATAFOG_ACCURACY_BACKEND=rust against tests/contracts/native-corpus.json,
which records the twelve known divergences with reasons. CI runs both
variants; pre-commit gains a ruff per-file-ignores entry for the dynamic
re-exports in datafog/native.py.
feat: native engine by default with Python fallback (4.10.0)
Core 1.4.0's text-mode email scanning trims label syntax by default, so
`email=alice@example.com` reports `alice@example.com` and quoted values drop
the wrapping quote. This moves the reviewed Rust contract file to
`rust-1.4.0.json` (no new deviations: every recorded case still matches,
verified by running the observation harness against 1.4.0), raises the
dependency floor, and resolves the `json-escaped` and `markdown-code-block`
native-corpus divergences.

`unicode-chinese-name` stays with a corrected reason: its corpus fixture
contains literal `?` characters, which are valid local-part characters, so the
reviewed expectation remains overlap-only.

Also stops the local verification gap this exposed: the two async/optional
dependency failures and one accuracy improvement are relative to an unmodified
main in the same environment (11 failed there, 10 here).
@sidmohan0
sidmohan0 merged commit 822b79b into dev Nov 16, 2026
37 of 65 checks passed
@sidmohan0 sidmohan0 mentioned this pull request Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant